Feed-forward Categorization of Body Expressions

Given the presence of massive feedback loops in brain networks, it is difficult to disentangle the contribution of feed-forward and feedback processing to the recognition of visual stimuli, in this case, of emotional body expressions. The aim of the present work is to shed light on how well feed-forward processing explains rapid categorization of this important class of stimuli. By means of parametric masking it may be possible to control the contribution of feedback activity in human participants. A close comparison is presented between human recognition performance and the performance of a computational neural model which exclusively modeled feed-forward processing and was engineered to fulfill the computational requirements of recognition. Results show that the longer the SOA (Stimulus Onset Asynchrony) the closer the performance of the human participants was to the values predicted by the model, with an optimum at an SOA of 100 ms. At short SOA latencies the human performance deteriorated, but the categorization of the emotional expressions was still above baseline. The data suggest that, although theoretically feedback arising from infero-temporal cortex is likely to be blocked when the SOA is 100 ms, human participants still seem to rely on more local visual feedback processing to equal the model’s performance. FEED-FORWARD CATEGORIZATION OF BODY EXPRESSIONS 3 A computational feed-forward model predicts categorization of masked emotional body language for longer, but not for shorter latencies Humans are capable of categorizing extremely quickly and accurately a wide variety of natural visual stimuli. Recent evidence suggests that this capability may be due to a fast feedforward processing stream involving brain networks specialized in certain types of stimuli (Fabre-Thorpe, Delorme, Marlot, & Thorpe, 2001). The aim of the present work is to shed some light on how well feed-forward processing explains rapid processing of an important class of stimuli, namely human body postures conveying emotion. To this end we compare a computational model of feed-forward categorization to a behavioral experiment in which the available processing time was carefully limited. In previous decades a number of research reports have focused on the processing of faces and their expressions in order to explore how we process emotions, and a number of computational models have been offered. More recently researchers have started to investigate the issue of bodily expression recognition. Switching to a new category can potentially provide evidence that human emotion theories may generalize to affective signals other than facial expressions (de Gelder, 2006, 2009). Results from a number of behavioural experiments using independent stimulus sets now allow us to conclude that recognition of emotions is similarly easy for face and body stimuli. Available literature has already firmly established that emotional bodily expressions clearly and rapidly convey the emotional, intentional and mental state of a person (Meeren, van Heijnsbergen, & de Gelder, 2005; Stekelenburg & de Gelder, 2004; Van den Stock et al, 2011) and that full awareness of the visual stimulus or intact striate visual cortex FEED-FORWARD CATEGORIZATION OF BODY EXPRESSIONS 4 are not essential (de Gelder, Vroomen, & Weiskrantz, 1999; Stienen, & de Gelder, 2011; Tamietto et al., 2009; Tamietto & de Gelder, 2010). Schindler, Van Gool and de Gelder (2008) have shown that a computational neural model which modeled exclusively feed-forward processes was capable of categorizing a set of seven different emotional bodily expressions in much the same way as human observers did. However, there was no time limit on the presentation of the bodily expressions in the human categorization task. Given the presence of massive feedback loops in brain networks, it is unclear whether human performance was only based on feedforward processes with no contribution from feedback processes. By controlling the contribution of feedback in human participants a closer comparison between the brain networks and the assumptions of the model is possible. Masking is one of the most widely used techniques for exploring unconscious processing of visual information in neurologically intact observers, and seems an excellent technique to control the contribution of feedback processes. For example, Esteves and Öhman (1993) found that short duration (e.g. 33 ms) presentations of happy and angry facial expressions, replaced immediately by a neutral face (mask) with a longer duration (e.g. 50 ms), are below the participants’ identification threshold. Lamme and Roelfsema (2000) and Lamme (2006) argue that a visual stimulus activates the visual cortex (striate and extrastriate) between 40 and 80 ms after presentation. Next, the infero-temporal cortex (IT) is feedforward-activated starting from 80 ms. Feedback signals from this area re-enter the visual cortex. Assuming 1 to 3 nodes that separate IT and visual cortex and a maximum firing rate of 100 Hz for cortical neurons (Rennie, Wright, & Robinson, 2000) the signal re-enters the visual cortex between 90-110 ms after the onset of the target. This means that FEED-FORWARD CATEGORIZATION OF BODY EXPRESSIONS 5 a mask could interfere with re-entrant signals arising from IT when presented less than 110 ms after presentation. In other words, it is increasingly more likely that feedback is possible from the infero-temporal cortex when the SOA (Stimulus Onset Asynchrony), and thus the processing time for the target, increases. Neurological evidence indicates that masking selectively disrupts re-entrant signals to V1. For example, Lamme, Zipser and Spekreijse (2002) showed that masking seemed to selectively interrupt the recurrent interactions between V1 and higher visual areas in the macaque monkey brain. Fahrenfort, Scholte and Lamme (2007) found in a human EEG study that when a texture-defined square was masked with an SOA of 16 ms, ERP’s typically associated with re-entrant processes were absent. No differences in bilateral occipito-temporal areas were found before 110 milliseconds while more posterior ERP’s triggered by seen stimuli started to differ from those triggered by unseen stimuli. However, the nature of the masking effect still remains a matter of discussion. The masking effect could be a consequence of imprecise temporal resolution starting as early as the retina, but possibly at cortical levels as well. This is called ‘integration masking’. Alternatively, the masking effect could arise by interruption of target processing in higher areas involved in object recognition, or in this case, bodily expression recognition (see e.g. review by Enns & Di Lollo, 2000). In our study we presented participants with masked emotional bodily expressions, using a parametric masking procedure to disentangle the contributions of feedback processing to their categorization performance. Five emotional expressions (including neutral) were presented to the participants while the onset between target and mask (SOA, Stimulus Onset Asynchrony) was FEED-FORWARD CATEGORIZATION OF BODY EXPRESSIONS 6 parametrically varied between 33 and 133 ms. The participants were instructed to categorize the emotion and use their intuition whenever they could not clearly see the target stimulus. The same set of stimuli was cross-validated using the neural model designed by Schindler et al (2008) and the outcomes were compared. In addition, the neural model was tested on mixtures (linear combinations) between the targets and the mask, in order to explore how the model performs on degraded images. It is expected that up to an SOA of 100 ms feedback processes arising from IT would be blocked by the mask. According to theory, full feedback should be possible when the SOA is 133 ms or longer. If human participants can categorize bodily expressions in the absence of information carried by feedback processes, then the model should predict the human performance when SOA latencies are 100 ms or shorter.